{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deep deterministic policy gradient\n",
    "\n",
    "DDPG is an off-policy, model-free algorithm, designed for environments where the\n",
    "action space is continuous. In the previous chapter, we learned how the actor-critic\n",
    "method works. DDPG is an actor-critic method where the actor estimates the policy\n",
    "using the policy gradient, and the critic evaluates the policy produced by the actor\n",
    "using the Q function.\n",
    "\n",
    "DDPG uses the policy network as an actor and deep Q network as a critic. One\n",
    "important difference between the DPPG and actor-critic algorithms we learned in\n",
    "the previous chapter is that DDPG tries to learn a deterministic policy instead of\n",
    "a stochastic policy.\n",
    "\n",
    "First, we will get an intuitive understanding of how DDPG works and then we will\n",
    "look into the algorithm in detail.\n",
    "\n",
    "\n",
    "## An overview of DDPG\n",
    "\n",
    "DDPG is an actor-critic method that takes advantage of both the policy-based\n",
    "method and the value-based method. It uses a deterministic policy $\\mu$ instead of\n",
    "a stochastic policy $\\pi$.\n",
    "\n",
    "We learned that a deterministic policy tells the agent to perform one particular\n",
    "action in a given state, meaning a deterministic policy maps the state to one\n",
    "particular action:\n",
    "\n",
    "$$a = \\mu(s)$$\n",
    "\n",
    "\n",
    "Whereas a stochastic policy maps the state to the probability distribution over the\n",
    "action space:\n",
    "\n",
    "$$ a \\sim \\pi(s)$$\n",
    "\n",
    "In a deterministic policy, whenever the agent visits the state, it always performs the\n",
    "same particular action. But with a stochastic policy, instead of performing the same\n",
    "action every time the agent visits the state, the agent performs a different action\n",
    "each time based on a probability distribution over the action space.\n",
    "\n",
    "Now, we will look into an overview of the actor and critic networks in the DDPG\n",
    "algorithm."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Actor\n",
    "\n",
    "The actor in DDPG is basically the policy network. The goal of the actor is to learn\n",
    "the mapping between the state and action. That is, the role of the actor is to learn the\n",
    "optimal policy that gives the maximum return. So, the actor uses the policy gradient\n",
    "method to learn the optimal policy.\n",
    "\n",
    "## Critic\n",
    "The critic is basically the value network. The goal of the critic is to evaluate the action\n",
    "produced by the actor network. How does the critic network evaluate the action\n",
    "produced by the actor network? Let's suppose we have a Q function; can we evaluate\n",
    "an action using the Q function? Yes! First, let's take a little detour and recap the use\n",
    "of the Q function.\n",
    "\n",
    "We know that the Q function gives the expected return that an agent would obtain\n",
    "starting from state s and performing an action a following a particular policy. The\n",
    "expected return produced by the Q function is often called the Q value. Thus, given a\n",
    "state and action, we obtain a Q value:\n",
    "\n",
    "* If the Q value is high, then we can say that the action performed in that state\n",
    "is a good action. That is, if the Q value is high, meaning the expected return\n",
    "is high when we perform an action a in state s, we can say that the action a is\n",
    "a good action.\n",
    "* If the Q value is low, then we can say that the action performed in that state\n",
    "is not a good action. That is, if the Q value is low, meaning the expected\n",
    "return is low when we perform an action a in state s, we can say that the\n",
    "action a is not a good action.\n",
    "\n",
    "\n",
    "Okay, now how can the critic network evaluate an action produced by the actor\n",
    "network based on the Q function (Q value)? Let's suppose the actor network\n",
    "performs a down action in state A. So, now, the critic computes the Q value of\n",
    "moving down in state A. If the Q value is high, then the critic network gives feedback\n",
    "to the actor network that the action down is a good action in state A. If the Q value is\n",
    "low, then the critic network gives feedback to the actor network that the down action\n",
    "is not a good action in state A, and so the actor network tries to perform a different\n",
    "action in state A.\n",
    "\n",
    "Thus, with the Q function, the critic network can evaluate the action performed by\n",
    "the actor network. But wait, how can the critic network learn the Q function? Because\n",
    "only if it knows the Q function can it evaluate the action performed by the actor. So,\n",
    "how does the critic network learn the Q function? Here is where we use the deep\n",
    "Q network (DQN). We learned that with the DQN, we can use the neural network\n",
    "to approximate the Q function. So, now, we use the DQN as the critic network to\n",
    "compute the Q function.\n",
    "\n",
    "Thus, in a nutshell, DDPG is an actor-critic method and so it takes advantage of\n",
    "policy-based and value-based methods. DDPG consists of an actor that is a policy\n",
    "network and uses the policy gradient method to learn the optimal policy and the\n",
    "critic, which is a deep Q network, and it evaluates the action produced by the actor."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Now that we have a basic understanding of how the DDPG algorithm works, let's\n",
    "go into further detail in the next section. We will understand how exactly the actor and critic networks\n",
    "work by looking at them separately with detailed math.__"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
